Virtual location selection for virtual content

ABSTRACT

A method for placing content in an augmented reality system. A notification is received regarding availability of new content to display in the augmented reality system. A confirmation is received that indicates acceptance of the new content. Three dimensional information that describes the physical environment is provided, to an external computing device, to enable the external computing device to be used for selecting an assigned location in the physical environment for the new content. Location information is received, from the external computing device, that indicates the assigned location. A display location on a display system of the augmented reality system at which to display the new content so that the new content appears to the user to be displayed as an overlay at the assigned location in the physical environment is determined, based on the location information. The new content is displayed on the display system at the display location.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Pat. App. No.62/865,756, filed Jun. 24, 2019, which is incorporated by reference.

FIELD

This specification generally relates to mixed reality (MR) devices.

BACKGROUND

A typical way to view content on a conventional device is to open anapplication that can display the content on a display screen (e.g., amonitor of a computer, smartphone, tablet, etc.). A user can navigate tothe application to view the content. Normally, when the user is lookingat the display screen of the display, there is a fixed format as to howthe content is displayed within the application and on the displayscreen of the display device.

SUMMARY

Innovative aspects of the subject matter described in this specificationrelate to placement of content in augmented reality (AR) systems.

With virtual reality (VR), augmented reality (AR), and/or mixed reality(MR) systems, an application can display content in a spatialthree-dimensional (3D) environment. In particular, a location of acontent item can be selected in response to receiving a notification ofan availability of a new content stream. Thus, according to one generalimplementation, a user can use a mobile device as a controller to selecta location for virtual display of the new content item. A representationof the user's environment can be displayed on the mobile device, and theuser can select a location within the representation as corresponding toa location in the user's physical environment at which to virtuallydisplay the new content. An indication of the user's selected locationcan be provided by the mobile device to the AR system. In response toreceiving location information from the mobile device, the AR system canvirtually display the new content stream so that the new content streamappears to the user as an overlay at the selected location in the user'sphysical environment.

According to implementations described, a notification is receivedregarding availability of new content to display in an augmented realitysystem. The augmented reality system is configured to present content ona display system so that the content appears to a user to be affixed atan assigned location in a physical environment of the user. Aconfirmation input is received that indicates acceptance of the newcontent. In response to receiving the confirmation input, threedimensional information that describes the physical environment isprovided, to an external computing device external to the augmentedreality system, to enable the external computing device to be used forselecting an assigned location in the physical environment for the newcontent. Location information is received, from the external computingdevice, that indicates the assigned location in the physical environmentfor the new content. A display location on the display system at whichto display the new content so that the new content appears to the userto be displayed as an overlay at the assigned location in the physicalenvironment is determined, based on the location information. The newcontent is displayed on the display system at the display location.

The described content placement approach is advantageous in that a usercan use a mobile device as a controller for an AR system. The mobiledevice can receive 3D information from the AR system and display arepresentation of the user's environment that enables selection ofvirtual display locations for new and existing content. A user canselect a location for virtual display of new content using therepresentation of the user's environment and the AR system can virtuallydisplay the new content as an overlay at a physical locationcorresponding to the location selected using the representation of theenvironment.

Other implementations of this aspect include corresponding systems,apparatus, and computer programs recorded on computer storage devices,each configured to perform the operations of the methods.

The details of one or more implementations of the subject matterdescribed in this specification are set forth in the accompanyingdrawings and the description below. Other features, aspects, andadvantages of the subject matter will become apparent from thedescription, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a scene of a room in which a head-mounted AR systemis used.

FIG. 1B illustrates an example mobile device user interface.

FIG. 1C illustrates an updated scene of a room in which a head-mountedAR system virtually displays a new content item.

FIG. 2 illustrates an example implementation of an AR system thatincludes mobile device integration.

FIG. 3 depicts a flow chart of a method for displaying content within anAR system.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1A illustrates a scene of a room 100 in which a head-mounted ARsystem 101 is used. The head-mounted AR system 101 may be a mixedreality device that presents to the user an interface for interactingwith and experiencing a mixed reality world. The mixed reality world caninclude computer-generated content and real world physical objects inthe user's physical environment. The head-mounted AR system 101 canprovide images of virtual objects intermixed with physical objects in afield of view of the user, for example.

The mixed reality world can be seen by the user through eye piece(s) ofthe head-mounted AR system 101. For example, a monocular eye view 102 asseen through a right eye piece 104 of the head-mounted AR system 101 isshown. The eye pieces (including the right eye piece 104) of thehead-mounted AR system 101 can be at least partially transparent andserve as “optical see-through” displays through which the user can viewreal objects in the user's physical environment. The user's physicalenvironment is the physical surroundings of the user as the user movesabout and views real world objects through the head-mounted AR system101. For example, the user's physical environment, as seen in themonocular eye view 102, includes a chair 106, a table 108, andbookshelves 110. The bookshelves 110 are cubicle shelves that include agrid of individual shelves, including a shelf 111.

The head-mounted AR system 101 can be configured to present to the user,through the eye pieces of the head-mounted AR system 101, virtualcontent that can be perceived as augmentations to physical reality. Forexample, the head-mounted AR system 101 can produce images of virtualobjects which are transposed onto partially transparent physicalsurfaces. For instance, virtual content 112 can be virtually displayed,through the eye pieces of the head-mounted AR system 101, on a topsurface 114 of the table 108, when the head-mounted AR system 101determines that the table 108 is within a field of view of thehead-mounted AR system 101. Virtual displaying of content, as describedherein, refers to displaying content on a display system or device (suchas the right eye piece 104), such that the content appears to the userto be displayed as an overlay at a particular three dimensional locationin the physical environment. The head-mounted AR system 101 can beconfigured such that when the user turns their head or looks up or down,display devices within the head-mounted AR system 101 continue to renderthe virtual content 112 so that it appears to remain affixed to the topsurface 114 of the table 108.

Virtual content may include or correspond to web pages, blogs, digitalpictures, videos, news articles, newsletters, or music, to name a fewexamples. Virtual digital content may correspond to content stored on astorage device that is accessible by the head-mounted AR system 101 orvirtual content may be a presentation of streaming content, such as alive video feed. Multiple items of virtual content may be virtuallydisplayed on multiple, different physical surfaces in the room 100, toincrease an overall workspace area for the user onto which virtualcontent may be displayed. Various planar surfaces may be designated asvirtual displays, for example.

The head-mounted AR system 101 can determine that a new content item isavailable for virtual display by the head-mounted AR system 101. Thehead-mounted AR system 101 can make an internal determination that newcontent is available and/or may receive a notice from an external systemthat new content is available. The head-mounted AR system 101 maysubscribe to a bot or another content source, for example.

The new content may be various types of pushed content. Pushed contentcan be content that can be rendered into the user's environment withoutthe user having to search for or select the content. For example, pushedcontent may include (a) notifications from various applications such asstock notifications, newsfeeds, etc.; (b) prioritized content, forexample, updates and notifications from social media applications, emailupdates, or messages from contacts; (c) messages targeting broad targetgroups and/or specific target groups; or d) other types of content orcontent streams.

In response to determining that a new content item is available forvirtual display, a notification 116 can be displayed to the user. Thenotification 116 indicates that a new stock chart content item isavailable. The notification 116 is also prompting the user to select alocation within the physical environment upon which to virtually displaythe new stock chart.

The notification 116 can be virtually displayed to the user by thehead-mounted AR system 101. The notification 116 can be shown on aparticular physical surface within the room 100, or can be displayed asif appearing “in space.” As another example, the notification 116 can bedisplayed within the monocular eye view 102 relative to one or moreedges of the eye piece 104. Other examples include the notification 116being displayed on a physical display of a computing device (not shown)or as an audio notification to the user (e.g., as played throughspeakers of the head-mounted AR system 101 or another connectedcomputing device).

The user can configure various settings for controlling the display ofthe notification 116 and other new content notices. For instance, theuser can configure a particular location (e.g., physical surface,physical computing device display) on which to virtually display newcontent notifications. As another example, the user can configurenotification frequency (e.g., display immediately, display periodically)and content type(s) for which to display notices.

The user can interact with the notification 116 to indicateacceptance/confirmation of the notification 116 and/or to initiateselection of a location (e.g., within the physical environment of theroom 100) on which to virtually display the new content or contentstream. The user can select a particular location in a variety of ways.One approach for selecting a location is by using a user interfacedisplayed on a mobile device of the user.

FIG. 1B illustrates an example mobile device user interface 140. Theuser interface 140 can be displayed on a user device 142 of a user inresponse to a user receiving a notification (e.g., the notification 116described above with respect to FIG. 1A) for a new content item that canbe displayed in an AR environment.

The user interface 140 displays a spatial three-dimensional (3D)environment that includes a 3D model 144 of the room 100 described abovewith respect to FIG. 1A. The 3D model 144 can be a simplified geometricmodel of the room 100. For example, the 3D model 144 can be a digital 3Dplanar map (e.g., a “mini map”) of the room 100.

The 3D model 144 can be a static preconstructed representation of theroom 100. As another example, the 3D model 144 can be generated in nearreal time based on information generated by the head-mounted AR system101 of FIG. 1A. For instance, the head-mounted AR system 101 cangenerate and send 3D model information that can be used by the userdevice 142 to display the 3D model 144. In some implementations, thehead-mounted AR system 101 can respond to changes in head-pose and/oreye movements of the user and can send updated 3D model information tothe user device 142, so that an updated 3D model 144 can be displayed inthe user interface 140 (e.g., with the updated 3D model 144 representinga view of the room 100 from the current perspective of the user).

The user interface 140, which can present a simplified, processed viewof the room 100, can include representations of physical objects in theroom 100. For example, the user interface 400 includes a chair object146, a table object 148, and a bookshelves object 150 corresponding tothe chair 106, the table 108, and the bookshelves 110, respectively. Theuser can zoom and pan the user interface 140 to focus on a particulararea of the 3D model 144.

The user device 140 can be used as a controller, control panel, and/ornavigator for the AR environment. For instance, the user can interactwith the user interface 140 for placement of virtual content. Existingvirtual content can be represented in the user interface 140 by anobject or icon. For instance, the virtual content 112 of FIG. 1A isrepresented in the user interface 140 as an icon 152. The user canselect the icon 152 and drag the icon 152 to another location within theuser interface 140. In response to a moving of the icon 152 to anotherlocation within the user interface 140, the virtual content 112 can bedisplayed as an overlay at a corresponding location within the ARenvironment as seen by the user using the head-mounted AR system 101.

In some implementations, certain surfaces of objects in the userinterface 140 are identified (by the user device 142 or the head-mountedAR system 101) as suggested surfaces for content placement. Forinstance, a chair back 154 of the chair object 146 can be shown in ahighlighted manner (e.g., in a particular color) to indicate that theback of the chair 106 is a recommended location for content. As anotherexample, certain shelves of the top two rows of the bookshelves object150 can be highlighted, such as shelves 156, 158, 160, and 162, toindicate that corresponding individual shelves of the bookshelves 110are suitable for placing content. The lower row of shelves of thebookshelves 110 might not be identified as suggested locations forcontent, since the head-mounted AR system 101 may determine that thelower row of shelves of the bookshelves 110 are not currently visible tothe user (e.g., the table 108 may block a view of the lower shelves). Ingeneral, suggested surfaces can be determined based on a variety offactors, including field of view with respect to current head-pose oreye gaze, surface contour, surface texture, surface size, type ofcontent that may be placed on the surface (e.g., type(s) of existing ornew content), or whether the surface is currently occupied by virtualcontent.

In some implementations, suggested surfaces are highlighted in responseto a notification of new content, as surfaces that may be amenable forvirtual display of the new content. For example, suggested surfaces canbe highlighted in the user interface in response to a notification 164that instructs the user to select a location for a new stock chartcontent item. The notification 164 corresponds to the notification 116and can be displayed automatically in the user interface 140 inconjunction with the display of the notification 116 within the view 102or in response to user interaction with (e.g., confirmation of) thenotification 116.

The user can, in response to the notification 164, select a surface of aparticular object in the user interface 140 as a location for virtualdisplay of the new stock chart. The user can select a suggested surface,some other surface, or some other location within the 3D model 144, suchas a location that represents a location “floating in space” in the room100. As shown by a finger 166 of the user, the user selects the shelf158 of the bookshelves object 150, to indicate that the shelf 111 of thebookshelves 110 is a selected location for virtual display of the newstock chart.

An indication of the selection of the shelf 158 of the bookshelvesobject 150 can be transmitted to the head-mounted AR system 101. Forexample, 3D information such as coordinates, region identifier, etc. ofthe shelf 158 can be sent to the head-mounted AR system 101.

In some implementations, in response to selection of the shelf 158 asthe location for virtual display of the new stock chart, a new iconrepresenting the new stock chart is displayed in the user interface 140,in/on the shelf 158. In some implementations, in response to (or inconjunction with) the display of the notification 164, an iconrepresenting the new stock chart is displayed at an initial/defaultlocation within the user interface 140. The user can move the new iconto a desired location, to finalize selection of the location for the newstock chart. The initial/default location can be, for example, thecenter of the user interface 140, a particular suggested surface, suchas a surface that has a highest suitability score, or a surface orlocation that is selected based on a configured user preference. If anew icon is displayed representing the new stock chart, the new icon canhave a different appearance (e.g., different color, flashing, otheremphasis) from the icon 152, at least until the user finalizes alocation for the new stock chart.

The user interface 140 can be configured such that when moving an iconto a particular surface, the icon “snaps” to the surface. As anotherexample, the user can move an icon by using a “drag and drop” operation,and in some cases, a user can confirm an end destination/location afterthe drag and drop operation has completed.

In response to selection of the shelf 158, the notification 164 can beremoved from the user interface 140. In some implementations, aselection confirmation notice (e.g., “stock chart location confirmed,”“thank you for selecting the location of the stock chart”) is displayedin response to selection of the shelf 158. After selection of the shelf158, the 3D model 144 can remain displayed in the user interface 140, toenable the user to perform other interactions with the AR environment,to receive future notifications of new content, etc.

FIG. 1C illustrates an updated scene of a room 180 in which ahead-mounted AR system 182 virtually displays a new content item. Theroom 180 corresponds to the room 100, and a view 184 seen through an eyepiece 186 of a head-mounted AR system 182 has been updated in responseto selection of a location at which to virtually display a new contentitem. For instance, the head-mounted AR system 182 can receiveinformation indicating user selection of the shelf 158 in the userinterface 140 of FIG. 1B.

The head-mounted AR system 182 can, in response to receiving informationindicating user selection of the shelf 158 as a location for a new stockchart content item, identify a shelf 188 of bookshelves 190 ascorresponding to the shelf 158 of the bookshelves object 150. Thehead-mounted AR system 182 can virtually display a virtual stock chartcontent item 192 in/on the shelf 188. For example, the virtual stockchart content item 192 can be rendered and superimposed substantiallyover the real world shelf 188, so that the virtual stock chart contentitem 192 appears to the user to be displayed as an overlay on top of theshelf 188.

The head-mounted AR system 182 can maintain a one to one mapping of theshelf 188 to the virtual stock chart content item 192. Using thehead-mounted AR system 182, the user can view the virtual stock chartcontent item 192 as if it appears on the mapped/matched shelf 188. Thevirtual stock chart content item 192 can be displayed to appear asphysically attached on the shelf 188, through projection and asperceived by the user.

FIG. 2 illustrates an example implementation of an AR system 200 thatincludes mobile device integration. A head-mounted AR system 202 can beconfigured to present to a user virtual content that can be perceived asaugmentations to physical reality. For example, an AR environmentgenerator 204 can provide images of virtual objects that can beintermixed with physical objects in a field of view of the user.

The head-mounted AR system 202 can receive a notification from a newcontent notifier 206 that a new content item is available for virtualdisplay by the head-mounted AR system 202. A new content subscriber 208can subscribe to and receive notifications from the new content notifier206, for example. As another example, the head-mounted AR system 202 canmake an internal determination that new content is available for virtualdisplay to the user.

The AR environment generator 204 can generate a virtual content item fordisplaying the new content by the head-mounted AR system 202. Thevirtual content item can be displayed by the head-mounted AR system 202so that it virtually appears to the user at a default location or thevirtual content item can be displayed so that it virtually appears tothe user at a user-selected location. The head-mounted AR system 202 cansend information to a mobile device 210, using a communication interface212, to enable the user to use the mobile device 210 to select alocation within the environment of the user for virtually displaying thevirtual content item. The AR environment Generator 204 can generate 3Dinformation about the environment of the user and the 3D information canbe sent to the mobile device 210 using the communication interface 212.The head-mounted AR system 202 can also send information about the newcontent item (e.g., a description of the content) and a request forselection of a location for the new content item.

The mobile device 210 can receive, using a communication interface 214,the 3D information about the user's environment and the informationabout the new content item. A 3D model renderer 216 of an AR controllerapplication 218 can render a 3D representation of the user's environmentwithin the AR controller application 218. The 3D representation caninclude representations of physical objects within the environment and,if applicable, representations of existing virtual content that arecurrently being virtually displayed by the head-mounted AR system 202 inthe user's environment.

The AR controller application 218 can display an instruction to the userto select a location for virtual display of the new content item. Acontent location selector 220 can enable the user to select a locationon the 3D representation of the user's environment, as corresponding toa location for virtual display of the new content item. The location canbe a surface of a rendered object that corresponds to a physical objectin the user's environment or the location can be 3D coordinates that mayor may not correspond to a physical object.

The AR controller application 218 can send, using the communicationinterface 214, location information for the selected location of the newcontent item to the head-mounted AR system 202. The location informationcan be 3D coordinates, region information, object information, or othertypes of information or identifiers. The AR environment generator 204can present or project the virtual content item so that the virtualcontent item appears to the user to be located at a location in theuser's physical environment that corresponds to the location informationreceived from the mobile device 210.

FIG. 3 depicts a flow chart of a method 300 for displaying contentwithin an AR system. A notification regarding availability of newcontent to display in an augmented reality system is received (310). Theaugmented reality system is configured to present content on a displaysystem so that the content appears to a user to be affixed at anassigned location in a physical environment of the user. Thenotification can be received from an external source that is external tothe augmented reality system. As another example, the notification canbe received as a result of an internal determination by the augmentedreality system that new content is available. The notification can bedisplayed on the display system.

A confirmation input that indicates acceptance of the new content isreceived (320). The confirmation input can be an interaction with thenotification, a voice command, or some other type of input. Thenotification can be removed from the display system in response toreceiving the confirmation input.

In response to receiving the confirmation input, three dimensionalinformation that describes the physical environment is provided to anexternal computing device external to the augmented reality system(330). The three dimensional information can be provided to enable theexternal computing device to be used for selecting an assigned locationin the physical environment for the new content. The external computingdevice can be a mobile device or some other type of computing device.

The three dimensional information can include three dimensionalinformation for candidate assigned locations for the new content. Thecandidate assigned locations can be locations that are determined to besuitable for overlaying a display of the new content. Candidate assignedlocations can correspond to physical surfaces in the physicalenvironment. The three dimensional information can include assignedlocations of existing content currently being displayed on the displaysystem.

The three dimensional representation can correspond to a staticrepresentation of the physical environment. As another example, thethree dimensional information can correspond to a current perspective ofthe user as seen through the display system. Updated three dimensionalinformation can be generated and provided to the external computingdevice in response to detection of a change in user perspective.

Location information that indicates the assigned location in thephysical environment for the new content is received, from the externalcomputing device (340). The location information can be 3D coordinates,region identifier(s), object identifiers, or some other type of locationinformation. The location information can correspond to a location theuser selects on a three dimensional representation of the physicalenvironment that is displayed on the external computing device.

A display location is determined, based on the location information(350). The display location is a location on the display system at whichto display the new content so that the new content appears to the userto be displayed as an overlay at the assigned location in the physicalenvironment.

The new content is displayed on the display system at the displaylocation (360).

Additionally or alternatively, an updated assigned location for a firstexisting content item can be received from the external computingdevice. An updated display location on the display system can bedetermined, based on the updated assigned location. The first existingcontent item can be displayed on the display system at the updateddisplay location, so that the first existing content item appears to theuser to be displayed as an updated overlay at the updated assignedlocation in the physical environment.

The described systems, methods, and techniques may be implemented indigital electronic circuitry, computer hardware, firmware, software, orin combinations of these elements. Apparatus implementing thesetechniques may include appropriate input and output devices, a computerprocessor, and a computer program product tangibly embodied in amachine-readable storage device for execution by a programmableprocessor. A process implementing these techniques may be performed by aprogrammable processor executing a program of instructions to performdesired functions by operating on input data and generating appropriateoutput. The techniques may be implemented using one or more computerprograms or non-transitory computer-readable storage media that includesinstructions that are executable on a programmable system including atleast one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice.

Each computer program may be implemented in a high-level procedural orobject-oriented programming language, or in assembly or machine languageif desired; and in any case, the language may be a compiled orinterpreted language. Suitable processors include, by way of example,both general and special purpose microprocessors. Generally, a processorwill receive instructions and data from a read-only memory and/or arandom access memory. Storage devices suitable for tangibly embodyingcomputer program instructions and data include all forms of non-volatilememory, including by way of example, semiconductor memory devices, suchas Erasable Programmable Read-Only Memory (EPROM), Electrically ErasableProgrammable Read-Only Memory (EEPROM), and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Anyof the foregoing may be supplemented by, or incorporated in, speciallydesigned ASICs (application-specific integrated circuits).

Computer-readable medium may be a machine-readable storage device, amachine-readable storage substrate, a memory device, a composition ofmatter effecting a machine-readable propagated signal, or a combinationof one or more of them. The term “data processing apparatus” encompassesall apparatus, devices, and machines for processing data, including byway of example a programmable processor, a computer, or multipleprocessors or computers. The apparatus may include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them. A propagated signal is anartificially generated signal, e.g., a machine-generated electrical,optical, or electromagnetic signal that is generated to encodeinformation for transmission to suitable receiver apparatus.

A computer program, also known as a program, software, softwareapplication, script, plug-in, or code, may be written in any form ofprogramming language, including compiled or interpreted languages, andit may be deployed in any form, including as a standalone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program does not necessarilycorrespond to a file in a file system. A program may be stored in aportion of a file that holds other programs or data in a single filededicated to the program in question, or in multiple coordinated files.A computer program may be executed on one computer or on multiplecomputers that are located at one site or distributed across multiplesites and interconnected by a communication network.

The processes and logic flows described in this specification may beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows may also be performedby, and apparatus may also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read only memory ora random access memory or both.

Elements of a computer may include a processor for performinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto optical disks,or optical disks. However, a computer may not have such devices.Moreover, a computer may be embedded in another device, e.g., a tabletcomputer, a mobile telephone, a personal digital assistant (PDA), amobile audio player, a VAR system, to name just a few. Computer-readablemedia suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto optical disks; and CD ROM and DVD-ROMdisks. The processor and the memory may be supplemented by, orincorporated in, special purpose logic circuitry.

While this specification contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular embodiments. Certain features that are described in thisspecification in the context of separate embodiments may also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment mayalso be implemented in multiple embodiments separately or in anysuitable sub-combination. Moreover, although features may be describedabove as acting in certain combinations and may even be claimed as such,one or more features from a claimed combination may, in some cases, beexcised from the combination, and the claimed combination may bedirected to a sub-combination or variation of a sub-combination. Forexample, although the mapping operation is described as a series ofdiscrete operations, the various operations may be divided intoadditional operations, combined into fewer operations, varied in orderof execution, or eliminated, depending on the desired implementation.

Similarly, the separation of various system components in theembodiments described above should not be understood as requiring suchseparation in all embodiments, and it should be understood that thedescribed program components and systems may generally be integratedtogether in a single software product or packaged into multiple softwareproducts. For example, although some operations are described as beingperformed by a processing server, one of more of the operations may beperformed by the smart meter or other network components.

Terms used herein and especially in the appended claims (e.g., bodies ofthe appended claims) are generally intended as “open” terms (e.g., theterm “including” should be interpreted as “including, but not limitedto,” the term “having” should be interpreted as “having at least,” theterm “includes” should be interpreted as “includes, but is not limitedto,” etc.).

Additionally, if a specific number of an introduced claim recitation isintended, such an intent will be explicitly recited in the claim, and inthe absence of such recitation no such intent is present. For example,as an aid to understanding, the following appended claims may containusage of the introductory phrases “at least one” and “one or more” tointroduce claim recitations. However, the use of such phrases should notbe construed to imply that the introduction of a claim recitation by theindefinite articles “a” or “an” limits any particular claim containingsuch introduced claim recitation to embodiments containing only one suchrecitation, even when the same claim includes the introductory phrases“one or more” or “at least one” and indefinite articles such as “a” or“an” (e.g., “a” and/or “an” should be interpreted to mean “at least one”or “one or more”); the same holds true for the use of definite articlesused to introduce claim recitations.

In addition, even if a specific number of an introduced claim recitationis explicitly recited, those skilled in the art will recognize that suchrecitation should be interpreted to mean at least the recited number(e.g., the bare recitation of “two recitations,” without othermodifiers, means at least two recitations, or two or more recitations).Furthermore, in those instances where a convention analogous to “atleast one of A, B, and C, etc.” or “one or more of A, B, and C, etc.” isused, in general such a construction is intended to include A alone, Balone, C alone, A and B together, A and C together, B and C together, orA, B, and C together. The term “and/or” is also intended to be construedin this manner.

The use of the terms “first,” “second,” “third,” etc., are notnecessarily used herein to connote a specific order or number ofelements. Generally, the terms “first,” “second,” “third,” etc., areused to distinguish between different elements as generic identifiers.Absent a showing that the terms “first,” “second,” “third,” etc.,connote a specific order, these terms should not be understood toconnote a specific order. Furthermore, absence a showing that the terms“first,” “second,” “third,” etc., connote a specific number of elements,these terms should not be understood to connote a specific number ofelements. For example, a first widget may be described as having a firstside and a second widget may be described as having a second side. Theuse of the term “second side” with respect to the second widget may beto distinguish such side of the second widget from the “first side” ofthe first widget and not to connote that the second widget has twosides.

What is claimed is:
 1. A computer-implemented method comprising:receiving a notification regarding availability of new content todisplay in an augmented reality system, wherein the augmented realitysystem is configured to present content on a display system so that thecontent appears to a user to be affixed at an assigned location in aphysical environment of the user; receiving a confirmation input thatindicates acceptance of the new content; in response to receiving theconfirmation input, providing, to an external computing device externalto the augmented reality system, three dimensional information thatdescribes the physical environment, to enable the external computingdevice to be used for selecting an assigned location in the physicalenvironment for the new content; receiving, from the external computingdevice, location information that indicates the assigned location in thephysical environment for the new content; determining, based on thelocation information, a display location on the display system at whichto display the new content so that the new content appears to the userto be displayed as an overlay at the assigned location in the physicalenvironment; and displaying the new content on the display system at thedisplay location.
 2. The computer-implemented method of claim 1, whereinthe external computing device is a mobile device.
 3. Thecomputer-implemented method of claim 1, wherein the notification isreceived from an external source that is external to the augmentedreality system.
 4. The computer-implemented method of claim 1, whereinthe notification is received as a result of an internal determination bythe augmented reality system that new content is available.
 5. Thecomputer-implemented method of claim 1, further comprising displayingthe notification on the display system.
 6. The computer-implementedmethod of claim 5, further comprising removing the notification from thedisplay system in response to receiving the confirmation input.
 7. Thecomputer-implemented method of claim 1, further comprising: determiningcandidate assigned locations in the physical environment for the newcontent; wherein the three dimensional information provided to theexternal computing device includes three dimensional information for thecandidate assigned locations.
 8. The computer-implemented method ofclaim 1, wherein determining the three dimensional informationcomprises: determining a current perspective of the user as seen throughthe display system; and generating the three dimensional informationbased on the current perspective.
 9. The computer-implemented method ofclaim 8, further comprising: determining that the current perspective ofthe user has changed to a new perspective; determining updated threedimensional information based on the new perspective; and providing theupdated three dimensional information to the external computing device.10. The computer-implemented method of claim 1, wherein the threedimensional information corresponds to a static representation of thephysical environment of the user.
 11. The computer-implemented method ofclaim 1, further comprising: determining existing content currentlybeing displayed by the display system; and determining assignedlocations for the existing content; wherein the three dimensionalinformation provided to the external computing device includes assignedlocation information for the existing content.
 12. Thecomputer-implemented method of claim 11, further comprising: receiving,from the external computing device, an updated assigned location for afirst existing content item; determining, based on the updated assignedlocation, an updated display location on the display system at which todisplay the first existing content item so that the first existingcontent item appears to the user to be displayed as an updated overlayat the updated assigned location in the physical environment; anddisplaying the first existing content item on the display system at theupdated display location.
 13. An augmented reality system, comprising: aframe structure comprising a display system configured to displaycontent so that the content appears to a user to be affixed at anassigned location in a physical environment of the user; and a processorcoupled to the frame structure, the processor configured to: receive anotification regarding availability of new content to display on thedisplay system; receive a confirmation input that indicates acceptanceof the new content; in response to receiving the confirmation input,provide, to an external computing device external to the augmentedreality system, three dimensional information that describes thephysical environment, to enable the external computing device to be usedfor selecting an assigned location in the physical environment for thenew content; receive, from the external computing device, locationinformation that indicates the assigned location in the physicalenvironment for the new content; determine, based on the locationinformation, a display location on the display system at which todisplay the new content so that the new content appears to the user tobe displayed as an overlay at the assigned location in the physicalenvironment; and display the new content on the display system at thedisplay location.
 14. The augmented reality system of claim 13, whereinthe external computing device is a mobile device.
 15. The augmentedreality system of claim 13, wherein the notification is received from anexternal source that is external to the augmented reality system. 16.The augmented reality system of claim 13, wherein the notification isreceived as a result of an internal determination by the augmentedreality system that new content is available.
 17. A non-transitorycomputer-readable storage medium comprising instructions, which, whenexecuted by one or more computer processors, cause the one or morecomputer processors to perform operations comprising: receiving anotification regarding availability of new content to display in anaugmented reality system, wherein the augmented reality system isconfigured to present content on a display system so that the contentappears to a user to be affixed at an assigned location in a physicalenvironment of the user; receiving a confirmation input that indicatesacceptance of the new content; in response to receiving the confirmationinput, providing, to an external computing device external to theaugmented reality system, three dimensional information that describesthe physical environment, to enable the external computing device to beused for selecting an assigned location in the physical environment forthe new content; receiving, from the external computing device, locationinformation that indicates the assigned location in the physicalenvironment for the new content; determining, based on the locationinformation, a display location on the display system at which todisplay the new content so that the new content appears to the user tobe displayed as an overlay at the assigned location in the physicalenvironment; and displaying the new content on the display system at thedisplay location.
 18. The non-transitory computer-readable storagemedium of claim 17, wherein the external computing device is a mobiledevice.
 19. The non-transitory computer-readable storage medium of claim17, wherein the notification is received from an external source that isexternal to the augmented reality system.
 20. The non-transitorycomputer-readable storage medium of claim 17, wherein the notificationis received as a result of an internal determination by the augmentedreality system that new content is available.